inamerrata

Continuing the theme from my previous post — the first and fundamental thing any distro needs to have, and thus the first and fundamental thing to think about disintermediating, is some way of distributing software. That might be burning CDs and sending them out to resellers or OEMs, or having your stuff available for download, whether by putting it up on your own server, hosting it via a dedicated high-reliability provider like akamai, or maintaining a network of mirror sites and keeping them up to date.

And, of course, anyone who wants to distribute their own software has to do the exact same thing — which either means doing it themselves, and not being able to take advantage of the scalability the distributor has already made work, or going through the distribution and dealing with the problems that entails, as well as the benefits.

In this case, disintermediation and decentralisation are pretty much one and the same thing: and decentralising content distribution is already a well understood problem: that’s what peer-to-peer is all about, and peer-to-peer distribution of, well, distributions is already very successful — at least when it comes to CD (and DVD) images. Which means that just about anyone can create a CD image, and distribute it to the world in exactly the same way a major organisation like Debian or Red Hat would — upload a torrent file, run a seed, and let the magic of BitTorrent do its thing. Scalability is no longer something you need a distribution organisation to manage, instead it’s built directly into the technology itself.

Unfortunately BitTorrent is designed for large static images — not something you update on a daily basis. And for better or worse, my preferred distribution (Debian testing, currently lenny) does update that frequently.

Fortunately, though, BitTorrent’s an open protocol and has dozens of open source implementations — so last year I was able to propose it as part of the Google Summer of Code, and was lucky enough to get a few responses and a student to work on it. It didn’t get much further than that unfortunately — the student lost internet access not long into the programme (and for most of its duration), and that was pretty much that. So when this year’s SoC rolled around, I didn’t really expect much, and didn’t put the idea up for consideration again, but lo and behold someone came through and asked if it was still possible and if there was any more information, and when I forwarded on some of the mails from the previous year we ended up with a second chance at the project.

So far it’s looking pretty good — we’ve had a lot more success at keeping in touch, and thanks to the extended schedule for the SoC this year we’ve been able to do a much better job of keeping on top of what’s going on. So much so, in fact, that there’s a first (alpha) release out before the SoC is officially due to start! Wonderful stuff!

What it does at the moment is allow you to take a Packages file (the stuff “apt-get update” downloads, which includes descriptions of all the packages that are available, how they inter-depend, and so forth), and from that information create a usable torrent from which to obtain the packages themselves which can then be used to share and distribute the packages.

There are two crucial steps in that: the first is allowing the torrent to work without requiring huge amounts of extra information to be distributed (which would introduce scalability problems of its own, instead of solving them), and the second is that the pieces that make up the torrent are selected in a way that matches the actual packages, so that when you upload a single new package, you are in fact only making a minor change to the torrent, rather than having it (on average) completely redefine half of the torrent (and again introduce scalability problems rather than solve them).

There’s more information on the DebTorrent wiki page and Cameron’s blog if you’re interested.

Anyway, it’s just an alpha release at the moment, which means while there’s code that does something, it’s not actually all that useful yet. The current plan is to next add some code to make it automatically prioritise packages based on what you’re actually using — so that rather than downloading all the debs it see, it’ll only download the ones you’ve got installed, or maybe only newer versions of the ones you’ve got installed, which should get us pretty close to the point where it’s actually useful for something.

The end result, of course, is to build a tool that you can point at a Debian archive, run it on a machine connected to the Internet, and you won’t have to do anything more to have a reliable, scalable and reasonably efficient means of allowing your users to distribute and update their systems. In this case, scalable means that if you end up with as many users as Debian or Ubuntu, your users will have a comparable experience, as if you’d arranged for a similarly comprehensive mirror network, without actually having to do the leg work.

And heck, presuming that works, it doesn’t even matter if no one else actually does that — it’s worth it even if it saves Debian or Ubuntu the effort of keeping track of a mirror network by hand.

There are interesting possibilities at smaller scales too, of course. :)